重量衰减通常用于确保具有批归归量的深神经网络的训练实践中的良好概括（BN-DNNS），在该训练中，由于归一化，某些卷积层对于重量重新恢复是不变的。在本文中，我们证明了重量衰减的实际用法仍然存在一些未解决的问题，尽管现有的理论工作在解释BN-DNNS中体重衰减的影响方面。一方面，当非自适应学习率例如使用动量的SGD，即使在初始训练阶段，有效学习率也会继续增加，从而导致许多神经体系结构的过度拟合效果。另一方面，在SGDM和自适应学习率优化器中，例如亚当，体重衰减对概括的影响对超参数非常敏感。因此，找到最佳的重量衰减参数需要广泛的参数搜索。为了解决这些弱点，我们建议使用简单而有效的重量重新缩放（WRS）方案来规范重量规范，以替代体重衰减。 WRS通过将重量标准明确地重新定为单位规范来控制重量规范，从而防止梯度增加，但也确保了足够大的有效学习率以提高概括。在各种计算机视觉应用程序中，包括图像分类，对象检测，语义细分和人群计数，我们与重量衰减，隐含重量重新缩放（重量标准化）和梯度投影（ADAMP）相比，显示了WR的有效性和鲁棒性。

嵌套辍学是辍学操作的变体，能够根据训练期间的预定义重要性订购网络参数或功能。它已被探索：I。构造嵌套网络：嵌套网是神经网络，可以在测试时间（例如基于计算约束）中立即调整架构的架构。嵌套的辍学者隐含地对网络参数进行排名，生成一组子网络，从而使任何较小的子网络构成较大的子网络的基础。 ii。学习排序表示：应用于生成模型的潜在表示（例如自动编码器）对特征进行排名，从而在尺寸上执行密集表示的明确顺序。但是，在整个训练过程中，辍学率是固定为高参数的。对于嵌套网，当删除网络参数时，性能衰减在人类指定的轨迹中而不是从数据中学到的轨迹中。对于生成模型，特征的重要性被指定为恒定向量，从而限制了表示学习的灵活性。为了解决该问题，我们专注于嵌套辍学的概率对应物。我们提出了一个嵌套掉落（VND）操作，该操作以低成本绘制多维有序掩码的样品，为嵌套掉落的参数提供了有用的梯度。基于这种方法，我们设计了一个贝叶斯嵌套的神经网络，以了解参数分布的顺序知识。我们在不同的生成模型下进一步利用VND来学习有序的潜在分布。在实验中，我们表明所提出的方法在分类任务中的准确性，校准和室外检测方面优于嵌套网络。它还在数据生成任务上胜过相关的生成模型。

使用重量衰减来惩罚神经网络中的重量规范，这是一种标准的培训实践，可以使网络的复杂性正常。在本文中，我们表明，包括重量衰减在内的一个正规化家族无效地惩罚具有正均匀激活功能的网络的固有权重规范，例如线性，relu和max-pool-pool函数。由于同质性，网络指定的功能是在层之间的重量尺度转移的不变性。无效的正规化器对这种转移敏感，因此使模型容量不正常，导致过度拟合。为了解决这一缺点，我们提出了一个改进的正规器，该正常化程序是体重尺度转移不变的，因此有效地约束了神经网络的内在规范。派生的正常化程序是网络输入梯度的上限，因此最大程度地降低了改进的正规器也使对抗性鲁棒性受益。还考虑了剩余连接，我们表明我们的正规器还形成了这种残留网络的输入梯度的上限。我们证明了我们提出的正常化程序在各种数据集和神经网络体系结构上的功效，以改善概括和对抗性鲁棒性。

A recent study has shown a phenomenon called neural collapse in that the within-class means of features and the classifier weight vectors converge to the vertices of a simplex equiangular tight frame at the terminal phase of training for classification. In this paper, we explore the corresponding structures of the last-layer feature centers and classifiers in semantic segmentation. Based on our empirical and theoretical analysis, we point out that semantic segmentation naturally brings contextual correlation and imbalanced distribution among classes, which breaks the equiangular and maximally separated structure of neural collapse for both feature centers and classifiers. However, such a symmetric structure is beneficial to discrimination for the minor classes. To preserve these advantages, we introduce a regularizer on feature centers to encourage the network to learn features closer to the appealing structure in imbalanced semantic segmentation. Experimental results show that our method can bring significant improvements on both 2D and 3D semantic segmentation benchmarks. Moreover, our method ranks 1st and sets a new record (+6.8% mIoU) on the ScanNet200 test leaderboard. Code will be available at https://github.com/dvlab-research/Imbalanced-Learning.

When using LiDAR semantic segmentation models for safety-critical applications such as autonomous driving, it is essential to understand and improve their robustness with respect to a large range of LiDAR corruptions. In this paper, we aim to comprehensively analyze the robustness of LiDAR semantic segmentation models under various corruptions. To rigorously evaluate the robustness and generalizability of current approaches, we propose a new benchmark called SemanticKITTI-C, which features 16 out-of-domain LiDAR corruptions in three groups, namely adverse weather, measurement noise and cross-device discrepancy. Then, we systematically investigate 11 LiDAR semantic segmentation models, especially spanning different input representations (e.g., point clouds, voxels, projected images, and etc.), network architectures and training schemes. Through this study, we obtain two insights: 1) We find out that the input representation plays a crucial role in robustness. Specifically, under specific corruptions, different representations perform variously. 2) Although state-of-the-art methods on LiDAR semantic segmentation achieve promising results on clean data, they are less robust when dealing with noisy data. Finally, based on the above observations, we design a robust LiDAR segmentation model (RLSeg) which greatly boosts the robustness with simple but effective modifications. It is promising that our benchmark, comprehensive analysis, and observations can boost future research in robust LiDAR semantic segmentation for safety-critical applications.

Proteins are fundamental biological entities that play a key role in life activities. The amino acid sequences of proteins can be folded into stable 3D structures in the real physicochemical world, forming a special kind of sequence-structure data. With the development of Artificial Intelligence (AI) techniques, Protein Representation Learning (PRL) has recently emerged as a promising research topic for extracting informative knowledge from massive protein sequences or structures. To pave the way for AI researchers with little bioinformatics background, we present a timely and comprehensive review of PRL formulations and existing PRL methods from the perspective of model architectures, pretext tasks, and downstream applications. We first briefly introduce the motivations for protein representation learning and formulate it in a general and unified framework. Next, we divide existing PRL methods into three main categories: sequence-based, structure-based, and sequence-structure co-modeling. Finally, we discuss some technical challenges and potential directions for improving protein representation learning. The latest advances in PRL methods are summarized in a GitHub repository https://github.com/LirongWu/awesome-protein-representation-learning.

Denoising Diffusion Probabilistic Models (DDPMs) are emerging in text-to-speech (TTS) synthesis because of their strong capability of generating high-fidelity samples. However, their iterative refinement process in high-dimensional data space results in slow inference speed, which restricts their application in real-time systems. Previous works have explored speeding up by minimizing the number of inference steps but at the cost of sample quality. In this work, to improve the inference speed for DDPM-based TTS model while achieving high sample quality, we propose ResGrad, a lightweight diffusion model which learns to refine the output spectrogram of an existing TTS model (e.g., FastSpeech 2) by predicting the residual between the model output and the corresponding ground-truth speech. ResGrad has several advantages: 1) Compare with other acceleration methods for DDPM which need to synthesize speech from scratch, ResGrad reduces the complexity of task by changing the generation target from ground-truth mel-spectrogram to the residual, resulting into a more lightweight model and thus a smaller real-time factor. 2) ResGrad is employed in the inference process of the existing TTS model in a plug-and-play way, without re-training this model. We verify ResGrad on the single-speaker dataset LJSpeech and two more challenging datasets with multiple speakers (LibriTTS) and high sampling rate (VCTK). Experimental results show that in comparison with other speed-up methods of DDPMs: 1) ResGrad achieves better sample quality with the same inference speed measured by real-time factor; 2) with similar speech quality, ResGrad synthesizes speech faster than baseline methods by more than 10 times. Audio samples are available at https://resgrad1.github.io/.

Crowdsourcing, in which human intelligence and productivity is dynamically mobilized to tackle tasks too complex for automation alone to handle, has grown to be an important research topic and inspired new businesses (e.g., Uber, Airbnb). Over the years, crowdsourcing has morphed from providing a platform where workers and tasks can be matched up manually into one which leverages data-driven algorithmic management approaches powered by artificial intelligence (AI) to achieve increasingly sophisticated optimization objectives. In this paper, we provide a survey presenting a unique systematic overview on how AI can empower crowdsourcing - which we refer to as AI-Empowered Crowdsourcing(AIEC). We propose a taxonomy which divides algorithmic crowdsourcing into three major areas: 1) task delegation, 2) motivating workers, and 3) quality control, focusing on the major objectives which need to be accomplished. We discuss the limitations and insights, and curate the challenges of doing research in each of these areas to highlight promising future research directions.

Fine-grained classification and counting of bone marrow erythroid cells are vital for evaluating the health status and formulating therapeutic schedules for leukemia or hematopathy. Due to the subtle visual differences between different types of erythroid cells, it is challenging to apply existing image-based deep learning models for fine-grained erythroid cell classification. Moreover, there is no large open-source datasets on erythroid cells to support the model training. In this paper, we introduce BMEC (Bone Morrow Erythroid Cells), the first large fine-grained image dataset of erythroid cells, to facilitate more deep learning research on erythroid cells. BMEC contains 5,666 images of individual erythroid cells, each of which is extracted from the bone marrow erythroid cell smears and professionally annotated to one of the four types of erythroid cells. To distinguish the erythroid cells, one key indicator is the cell shape which is closely related to the cell growth and maturation. Therefore, we design a novel shape-aware image classification network for fine-grained erythroid cell classification. The shape feature is extracted from the shape mask image and aggregated to the raw image feature with a shape attention module. With the shape-attended image feature, our network achieved superior classification performance (81.12\% top-1 accuracy) on the BMEC dataset comparing to the baseline methods. Ablation studies also demonstrate the effectiveness of incorporating the shape information for the fine-grained cell classification. To further verify the generalizability of our method, we tested our network on two additional public white blood cells (WBC) datasets and the results show our shape-aware method can generally outperform recent state-of-the-art works on classifying the WBC. The code and BMEC dataset can be found on https://github.com/wangye8899/BMEC.

With the development of natural language processing techniques(NLP), automatic diagnosis of eye diseases using ophthalmology electronic medical records (OEMR) has become possible. It aims to evaluate the condition of both eyes of a patient respectively, and we formulate it as a particular multi-label classification task in this paper. Although there are a few related studies in other diseases, automatic diagnosis of eye diseases exhibits unique characteristics. First, descriptions of both eyes are mixed up in OEMR documents, with both free text and templated asymptomatic descriptions, resulting in sparsity and clutter of information. Second, OEMR documents contain multiple parts of descriptions and have long document lengths. Third, it is critical to provide explainability to the disease diagnosis model. To overcome those challenges, we present an effective automatic eye disease diagnosis framework, NEEDED. In this framework, a preprocessing module is integrated to improve the density and quality of information. Then, we design a hierarchical transformer structure for learning the contextualized representations of each sentence in the OEMR document. For the diagnosis part, we propose an attention-based predictor that enables traceable diagnosis by obtaining disease-specific information. Experiments on the real dataset and comparison with several baseline models show the advantage and explainability of our framework.